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(54) POINTING DEVICE AND INPUT METHOD 

(57)Abstract: 

PURPOSE: To input an operation command with a simple 
operation while the plant operator is moving or refer 
plural display images. 

CONSTITUTION: A mark 1101 is provided on a display 
screen 1001 or its peripheral part. Or a frame of a 
display screen itself is used for a mark and a visual 
position on the display screen 1001 is specified based on 
the information from an eye camera 1401 as to a visual 
point and information of the mark 1101 picked up by a 
visual camera 1501. Thus, the operator attains pointing 
operation by having only to view a point desired to be 
pointed out and the operability is improved. 
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* NOTICES * 

JPO and INPIT are not responsible for any 
damages caused by the use of this translation. 

1. This document has been translated by computer. So the translation may not reflect the 
original precisely. 

2. **** shows the word which can not be translated. 
3. In the drawings, any words are not translated. 



CLAIMS 

[Claim(s)] 

[Claim 1]While providing an identification device for identifying the display screen 
concerned in a part or a periphery of a display screen, A pointing device pinpointing a view 
position of a user who saw with coordinates on a display screen from a kind of the above- 
mentioned identification device and position information which a visual field camera which a 
user carried caught, and position information on a viewpoint of an eye camera which a user 
carried. 

[Claim 2]While providing an identification device for identifying the display screen 
concerned in a part or a periphery of a display screen, A pointing device pinpointing a view 
position of a user who saw with coordinates on a display screen from a kind of the above- 
mentioned identification device and position information which a visual field camera which a 
user carried caught, and position information on a light spot which light from a floodlight 
which a user carried forms on a display screen. 

[Claim 3]The pointing device according to claim 1 or 2 adding a function to specify a display 
screen in which a means to detect a direction which a position in which a user is, or a user 
has turned to was formed, and which a user has countered by this means when there are 
two or more display screens. 

[Claim 4]Claims 1 and 2 displaying a light spot position which said specified viewpoint of a 
user or a user points out on the position concerned on the display screen concerned by a 
marker, or a pointing device given in either of 3. 

[Claim 5]The pointing device according to any one of claims 1 to 4, wherein said 
identification device is a punctiform mark which emits light in a punctiform mark formed 
around the display screen in paints, visible light, or infrared rays. 

[Claim 6]The number of a mark punctiform [ said ], a color, a luminous wavelength, or the 
pointing device according to claim 5 characterized by identifying distinction of two or more 
display screens, or four directions of one display screen with a blinking period of a 
punctiform mark which emits light. 

[Claim 7]The pointing device according to any one of claims 1 to 4, wherein said 
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identification device is a photogen of the shape of the display screen circumference itself, 
infrared light provided around the display screen as a line frame, or visible light. 
[Claim 8]As opposed to each portion when it divides into some, a photogen which 
constitutes said line frame to each line frame of two or more display screens with 
wavelength or a blinking period of luminescence. The pointing device according to claim 7 
identifying distinction on two or more display screens, or four directions of one display 
screen. 

[Claim 9]The pointing device according to any one of claims 1 to 4, wherein said 
identification device is the mark attached to a specific figure or a specific figure currently 
displayed on a display screen. 

[Claim 10]ln a pointing device of any one statement of claim 1 thru/or 9, While specifying an 
object displayed on a light spot position which a viewpoint of said specified user or a user 
points out as a specification object from a position pinpointed [ above-mentioned ] and 
image data about the above-mentioned object, An input method, wherein a computer 
executes a processing instruction from an input means established independently to the 
above-mentioned specification object. 

[Claim 11]They are made into a specification object candidate when there are two or more 
objects displayed on a light spot position which said specified viewpoint of a user or a user 
points out, The input method according to claim 10 specifying said specification object from 
the above-mentioned specification object candidate with reference to an input through 
information about a display object prepared for others, and an input means from a user. 
[Claim 12]When a light spot position which said specified viewpoint of a user or a user 
points out is in a range on a display screen specified beforehand, Or the input method 
according to claim 10 or 1 1 characterized by a computer executing said command only 
when said specified specification object is one of the object groups defined beforehand. 
[Claim 13]The input method according to any one of claims 10 to 12, wherein said input 
means is one side or both sides of a means which takes out a command from a sound 
which a keyboard and a user utter. 

[Translation done.] 
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DETAILED DESCRIPTION 

[Detailed Description of the Invention] 
[0001] 

[Industrial Application]Especially this invention relates to the pointing device which outputs 
the signal which specifies the apparatus or device, when the apparatus or the device 
displayed on the display screen is exposed to human being's viewpoint or beam of light with 
respect to a pointing device (a pointing device is only called hereafter). 
[0002] 

[Description of the Prior Art]As a conventional pointing device, there are JP.60-85899.A or 
the Institute of Electronics, Information and Communication Engineers paper magazine A, a 
J71 -A volume, and a thing currently discussed in 2192-2197 pages of 12 No. In this 
conventional technology, it is asking for the user's point of regard using the eye camera or 
the eye camera, and the visual field camera, and can be used also for the pointing device 
on a screen. As other conventional technologies, by what was shown, for example in JP.62- 
196723.A. He is trying to amend the position of the eyeball which provided the point light 
source in the display screen periphery, detected angle of rotation of a user's head based on 
the picture about the point light source reflected to the visual field camera, and was 
detected by the eye camera. In what was shown in JP,62-53630,A and JP,62-53631,A, 
feature objects are installed in measuring space and the view information from an eye 
camera is amended based on the position of the feature objects within a view image. In 
these inventions, amendment of view information is possible also at the time of movement 
of a user position, pointing while the thing of the statement is known [ Japanese Patent 
Application No. / No. 99532 / 60 to ] as art which uses a pointing signal for pointing and a 
user moves is possible. 
[0003] 

[Problem(s) to be Solved by the lnvention]Although the thing shown in JP.60-85899.A and 
the Institute of Electronics, Information and Communication Engineers paper magazine of 
the above-mentioned conventional technology and the thing shown in Provisional 
Publication No. No. 196723 are effective in asking for a user's view position under the 
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situation where an eyeball and a head move, It cannot be used when the position of a 
user's body moves greatly. What was shown in JP,62-53630,A and 62-53631, An early 
view image and the last view image are memorized, since it is what amends a user's head 
position from it and the newest view image, a user's motion is large and there is a problem 
that processing becomes complicated, about a case so that feature objects may separate 
temporarily from the inside of a view image. When there are two or more display screens, 
the consideration about what it is specified for in which portion of which display screen a 
viewpoint is is not made. What was shown in Japanese Patent Application No. No. 99532 
[ 60 to ] needs to arrange many photo detectors around a screen, and there was a problem 
that it could not manufacture simple, especially about a large-sized display screen. 
[0004]The purpose of this invention is to provide a simple pointing device usable also when 
there are two or more display screens like [ in the case of a plant operation ] so that use 
while a user moves is possible. 
[0005] 

[Means for Solving the Problem]ln order to attain the above-mentioned purpose, in a 
pointing device which becomes this invention. Provide a mark in a display screen or its 
periphery, or a display screen's own frame is used as a mark, A view position on a display 
screen is pinpointed from information on the above-mentioned mark caught with information 
about a viewpoint from an eye camera, and a visual field camera, In the case of a pointing 
device by pointing, light from a luminescence implement attached to the above-mentioned 
mark, a fingertip, etc. is taken in with a monitor camera, and it specified a point which a 
user pointed at based on those signals. 
[0006] 

[Function]Since the position of a display screen is easily detectable when a visual field 
camera or a monitor camera detects the mark of the display screen circumference whether 
a user moves or there are two or more display screens and, From the position of the 
viewpoint of an eye camera, or the light spot by pointing, and a display screen position, it 
can specify which position (what existing) on a display screen is specified. And installation 
and its detection of the above-mentioned mark can be performed easily, and a special 
device etc. are not needed. 
[0007] 

[Example]Hereafter, an example explains this invention. Drawing 1 is a block diagram 
showing one example of this invention, and the infrared lamp 1101 is formed in four 
surrounding corners of the display screen 1001 of the display 11. The drawing of the 
tabulated drawing data 1301 is displayed on the display screen 1001 via the display control 
1201. The user 2401 has carried the visual field camera (thing to infrared rays which has 
good sensitivity) 1501 which is a small camera for photoing the situation of the transverse 
plane of the eye camera 1401 and user who detect a motion of an eyeball, and the 
microphone 1901 for voice input. The keyboard 1801 is prepared as other input devices. 
The camera-signal-processing device 1601 searches for the position coordinate of the 
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viewpoint in the display screen 1001 based on the signal of the eye camera 1401 and the 
visual field camera 1501. Based on this signal, the object specifying device 1701 specifies 
the object in the display screen at which the user is gazing from indication screen data on 
display to the display screen 1001. On the other hand, in the input analysis device 2001, 
the input from the microphone 1901 and the keyboard 1801 is analyzed using the data 
2101 for input analysis. This result is inputted into the input signal synthesis processing unit 
2201 , and is outputted to the computer 2301 as a processing instruction together with the 
information about the gaze subject from the object specifying device 1701 . 
[0008]Drawing_2_is a flow chart which shows operation of the camera-signal-processing 
device 1601. In this device, this processing is always performed periodically. First, the 
signal from the visual field camera 1501 and the eye camera 1401 is incorporated (Step 
9111), and then the position coordinate of the infrared lamp within a view, i.e., the monitor 
display of a visual field camera, is searched for from the signal of the visual field camera 
1501 based on a luminance level (Step 9112). Subsequently, the position coordinate of the 
viewpoint in the monitor display of the visual field camera 1501 is searched for from the 
signal of the eye camera 1401 (Step 9113). Furthermore, from the position coordinate and 
view position coordinates of an infrared lamp in monitor display, based on information, 
including the infrared lamp position of a actual display screen, etc., the usual image 
processings technique, such as transparent transformation, are used, and the coordinates 
of the viewpoint on the display screen 1001 are searched for (Step 9114). 
[0009] Drawing 3 is a flow chart which shows operation of the object specifying device 1 701 . 
The coordinates of the viewpoint first detected with the camera-signal-processing device 
1601 are incorporated (Step 9121), and the subject currently displayed on the view position 
with reference to indication screen data is specified based on the coordinates of this 
viewpoint (Step 9122). 

fOOIOI Drawing 4 is a figure showing an example of the tabulated drawing data 1301, and 
comprises the name of a display thing, and the coordinates on the display screen 1001 as 
which it is displayed. However, coordinates are coordinates of each vertex when the field 
where the display thing is displayed is approximated with a quadrangle. It is also possible to 
approximate a described area by a polygon, and it is necessary to prepare the coordinate 
data which balanced the number of each of that vertex in that case. 
[001 1]Next, operation of the input analysis device 2001 and the input signal synthesis 
processing unit 2201 is explained. These devices operate, only when there is an input from 
a microphone or a keyboard. 

[0012] Drawing 5 is a flow chart which shows operation of the input analysis device 2001 . 
This analyzes the input sentence from the keyboard 1801 or the microphone 1901 , 
changes and outputs it to a processing instruction, i.e., a function, and an argument value, 
and explains operation as an input sentence by making into an example the case where 
"ATSURYOKU trend HYOUJI" is inputted at Step 101. With reference to the part-of-speech 
category and the grammar 201 of a word which were written in the dictionary 202, the 
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morpheme and syntax analysis of an input sentence are conducted. As a result, division 
writing of the input sentence is carried out, and a semantic category and a meaning are 
added to each word, respectively. Here, as the technique of a morpheme and syntax 
analysis, the technique of normal use being carried out, for example, the prediction bottom- 
up method etc., is used. Grammar is a rule about the structure of a sentence, for example, 
is S(sentence) ->NP (noun phrase) S (sentence). 

The rule of getting S of ** and the left side being constituted by NP and S of the right-hand 
side is expressed. The display word, the semantic category, and four information on a 
meaning that the part-of-speech category and the entry were changed into the kana 
Chinese character are given to the dictionary about each entry. However, about the 
particle, only two, a part-of-speech category and a display word, are given. Authorization of 
the word included in an input sentence can be similarly carried out by consultation of a 
dictionary by the longest match principle usually used with a word processor etc. 
[0013]Next, the processing step 103 for identification of the contents of a command is 
performed. In this processing, the result of syntax analysis is incorporated and the contents 
of a command are identified using the command identification rule 203. A command needs 
SHOW-SVAR (presentation command of quantity of state) etc. here, for example. A 
command identification rule gives the standard sentence pattern for identifying the 
command to each command. In command identification, matching with an input sentence 
and each sentence pattern is tried one by one, and suppose that it is an input command the 
command corresponding to the matched sentence pattern so that drawing 6 may describe 
in detail. In the example of the figure, the sentence pattern "\<quantity of state> ? 
presentation + Carry out" matches with an input sentence, and the identified results that an 
input command is SHOW-SVAR are obtained. In this processing, in order to give a 
sentence pattern, some signs are prepared. Only when an input sentence has a 
particle"%", it is a sign showing the word used for matching, and it is possible to deal with 
the abbreviation of the particle often looked at by the dialogue sentence by this. "+" is a 
sign showing an omissible word and is because corresponding to a substantives stop etc. 
"?" is a sign showing the word of an arbitrary individual, and makes setting out of the 
sentence pattern easy by this. It means that T carries out matching with the semantic 
category of a word. 

[0014]The detailed contents of processing of the contents identification part of a command 
are explained here. Drawing 6 is the flow chart and sets a sentence pattern number as 1 
first. About the sentence pattern, i showing the number of the word in a sentence pattern is 
set as the total I of the word of a sentence pattern, j which shows the number of the word of 
an input sentence is set as the total J of the word of an input sentence (Step 401). Next, the 
i-th word Pi of the k-th sentence pattern and the word Wj of No. j of an input sentence are 
brought (Step 402). Conducts matching described below about this Pi and Wj is carried out. 
Since i is set as I and j is set as J, a word will be compared toward a beginning of a 
sentence in the beginning from the sentence end of an input sentence and a sentence 
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pattern. 

[0015]Processing of matching changes with existence of the form of the word Pi of a 
sentence pattern, i.e., the above-mentioned sign, (Step 403). First, processing when the 
sign does not stick is explained. In this case, Pi is compared with Wj (Step 404), and when 
these are the same, only 1 reduces i and j (Step 405). That is, let the word to compare be a 
thing in front of one. Since it is a case where all the words of a sentence pattern match with 
the word of an input sentence when i is set to 0 as a result (Step 406), the command name 
corresponding to that sentence pattern k is outputted noting that it is the contents of a 
command into which it was inputted (Step 410). 

[0016]On the other hand, when i is one or more (Step 406), the judgment about j is carried 
out (Step 407). As a result, if j is one or more, the processing from Step 402 will be 
repeated. Conversely, since matching with the sentence pattern k and an input sentence 
went wrong when j was 0, matching with the following sentence pattern (k+1) is prepared 
(Step 408). Subsequently, if the sentence pattern still remains (Step 409), processing from 
Step 402 will be performed about a new sentence pattern. On the other hand, when a 
sentence pattern is lost, it outputs that command identification is improper, and processing 
is ended (Step 412). 

[0017]When the sign T is attached to Pi, the semantic category of Pi and Wj is compared 
(Step 413). According to the result, the same processing as the above-mentioned is carried 
out. 

[0018]When the sign "%" is attached to Pi, it is judged whether the part-of-speech category 
of Wj is a particle (Step 414). As a result, if it is a particle, Pi will be compared with Wj (Step 
415) and the same processing as the above-mentioned will be carried out according to a 
result. On the other hand, when Wj is not a particle, j is reduced one, and if j is one or more 
(Step 417), it will return to Step 402. The processing about the following sentence pattern, 
i.e., the processing from Step 408, is carried out noting that matching with the sentence 
pattern k and an input goes wrong, since the word of the input sentence was lost on the 
other hand when j was 0. 

[0019]When the sign "+" is attached to "Pi", Pi is compared with Wj (Step 418), and if the 
same, processing from Step 405 will be carried out. On the other hand, in differing, if i is 
one or more after reducing i one, it will return to Step 402. Since I hear that all (except for 
an omissible thing) the words of the sentence pattern k were included in the input sentence 
when i was set to 0, it returns to Step 410 and command identification is ended. 
[0020]When Pi is a sign "?", word Pi-1 in front of "?" is compared with Wj (Step 421). As a 
result, if a word matches, 1 will be subtracted from i and conducts matching about the 
following word will be performed from Step 405. On the other hand, if not the same, j will be 
reduced one, and if j is one or more (Step 424), conducts matching will be carried out using 
the word in front of [ of an input sentence ] one from Step 402. Since matching with the 
sentence pattern k went wrong when j was 0, it performs from the conducts matching 408 
with the following sentence pattern k+1, i.e., a step. 
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[0021]After the command identification processing of Step 103 of drawing 5 explained in full 
detail above finishes, processing for acquisition of information required for execution of the 
command identified at Step 104 next is performed. That is, a required information item and 
its acquisition method were prepared as data of frame form as a method for controlling the 
information acquisition depending on a command, since some information is required 
because of the execution. This is called case frame. The case frame is defined for every 
command and has given the item of information required for execution of the command to 
each slot of the frame of a command name. It is shown by the example of the case frame 
204 of drawing 5 to command SHOW-SVAR that "quantity of state-, a -display style-, and 
three information on a "place- are required. Data required to acquire the information is given 
to the facets and the value of each slot. The name of the function which should start when 
there is no word which needs the semantic category name of the word which should be 
searched from an input sentence in an input sentence in IF-NEEDED facets is conferred 
upon VALUE facets. Here, the function in which the function STACK refers to a 
conversation history stack, and function ASK are functions which require the additional 
input of information of a questioner. It is for the function CAMERA using the result from the 
object specifying device 1701 . Starting of this function will write in ?Place as a value of a 
corresponding rank. 

[0022]The flow of processing of Step 104 is shown in the flow chart of drawing 7 . The case 
frame which has a command name as a frame name first is taken out (Step 501). Next, i is 
set as 1 (Step 502). After this initial processing, the value of the VALUE facets of the i-th 
slot is read, and it sets up as a value of Vi (Step 503). If the value of this Vi is not nil (Step 
504), the word which has a semantic category out of an input sentence will be looked for 
(Step 505). If the word acquired when the applicable word could be discovered (Step 506) 
was set up as a value of the information item of the i-th slot (Step 507) and the slot still 
remains (Step 510), i will be increased one (Step 511) and it will shift to the processing 
about the following slot. On the other hand, when a word cannot be discovered at Step 506, 
or when the value of Vi is nil at Step 504, the value of IF-NEEDED facets is read and it sets 
up as a value of nickel (Step 508). Subsequently, sequential execution of the function of 
nickel is carried out. As a result, when a value is obtained, it is set up as a value of the 
information item of the i-th slot (Step 509), and it shifts to the processing about the following 
slot. 

[0023]When the above operation is described about the example of the input of drawing 5 , 
there are "quantity of state-, -display style-, and three slots of a "place" in the case frame 
corresponding to command SHOW-SVAR. About "quantity of state", a word with the 
semantic category <quantity of state> first given as a value of VALUE facets is searched in 
an input sentence. Now, the corresponding word "pressure" is included in the input 
sentence. Thereby, a "pressure" is set up as a value of "quantity of state." Similarly, about a 
display style, a "trend" is obtained from an input sentence as a word with a semantic 
category <display style>, and it is set up as a value. Although the word of a semantic 
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category <Place> is searched from an input sentence about a "place", there is no word 
[ be / it / under / input sentence / correspondence ] now. Therefore, the value of IF- 
NEEDED facets is referred to and the function CAMERA is performed. "?Place" is set up as 
a value of this result "place." 

[0024]After acquisition of information required for an instruction execution finishes as 
mentioned above, processing for matching with the function and argument of Step 105 is 
performed by the last. In this processing, the value of the command obtained by pre- 
processing and an information item is matched with the function and argument of a 
program, and is outputted. In the example of the input of drawing 5 , a function name is set 
as a "trend display" from SHOW-SVAR and a "display style" trend. A "pressure" and "? 
Place" are set up as an argument value. 

r0025] Drawing 8 is a flow chart which shows operation of the input signal synthesis 
processing unit 2201. With this device, the function name and argument value which were 
appointed with the input analysis device 2001 are read first (Step 9151). Next, it is judged 
whether variable ?Place is in an argument (Step 9152). As a result, if there is no variable, it 
will output to a computer by making the function from the input analysis device 2001, and 
an argument value into a processing instruction (Step 9153). On the other hand, when 
there is a variable, the subject at which the user is gazing is read from the object specifying 
device 1701 (Step 9154). The name of a subject is set as variable ?Place, a processing 
instruction is completed, and it outputs to a computer (Step 9155). 
[0026]The signal about the viewpoint from the eye camera which the user carried when 
using the device which becomes this example, as explained above, and a visual field 
camera, And the subject currently displayed on the coordinates in the display screen at 
which the user is gazing, and there can be searched for using the signal about the picture 
of an infrared lamp provided in the periphery of the display screen. By this, pointing which 
does not have a service level limited becomes it is simple and possible. The input from a 
sound and a keyboard and the information about a gaze subject can be synthesized, and it 
can be considered as a command to a computer. By these, simplification of the input 
operation not only to pointing operation but a computer, improvement in workability, etc. are 
realizable. 

[0027]ln the example described above, although an infrared lamp shall be installed in four 
corners of a display screen as a mark for giving the shape of a display screen, various 
modification is possible for this. Drawing 9 is the example, infrared rays are sent to the 
frames 2701-2704 of four disclosure fibers from the sources 2501-2504 of infrared ray 
emission, each source of infrared ray emission blinks further, and the blinking periods differ 
by each light source. In using such a mark, in the camera-signal-processing device 1601, it 
asks from the blinking period of the mark to which a viewpoint takes a viewpoint and 
surrounds whether it is within the limit surrounded by which mark from the signal about the 
mark and viewpoint in the picture of a visual field camera. Then, if the coordinates of the 
viewpoint of the specified within the limit are searched for, the coordinates of the viewpoint 
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on a display screen can be searched for with sufficient accuracy. Especially this is effective 
when using a display like a large screen display. 

f0028] Drawing 10 is a mimetic diagram showing another example of a mark. The infrared 
rays from the one source 2505 of infrared ray emission are led to the point light sources 
2801-2804 via the star coupler 2601, the disclosure fiber 2700, and an optical fiber. The 
display screen 1001 is quadrisected and the number of the point light sources differs in the 
each. Use of this mark will judge whether a viewpoint is located in which region division 
from the number of the point light source in the same region division as a viewpoint to be 
the light from a disclosure fiber. About the other processing, it is the same as that of the 
example of drawing 9 . According to this example, the coordinates accuracy of a viewpoint 
can be improved and also it is effective in that one piece may be sufficient as the number of 
the infrared light source to be used. 

[0029]Although the mark quadrisected the display screen 1001 in these examples, it is 
possible to change the number of partitions. A mark cannot be attached about all the 
display screens, but a mark can be attached to a part of display screen, and only the 
portion can also be used as a field in which pointing is possible. For example, some 
screens are surrounded with a disclosure fiber, only when a viewpoint goes into this 
portion, the input in a sound is received, and it can be used for switching of voice input. 
Although the infrared light source was considered as a light source here, the function 
completely same also as the light source of other wavelength, for example, a light source of 
visible light, can realize this. 

f0030] Drawing 1 1 is a block diagram showing the 2nd example of this invention, the three 
display screens 1001-1003 are installed in the indicator 2901, and red and the green and 
blue marks 3001-3003 are attached to the periphery, respectively. The drawing of the 
tabulated drawing data 1302 is displayed on each display screen by the display control 
1202. The user 2401 carries the eye camera 1401, the visual field cameras 1501 and 1502, 
and the microphone 1901 for voice input, and the keyboard 1801 is also prepared. It asks 
for in which coordinates of which display screen the camera-signal-processing device 1602 
has a user's viewpoint based on the signal of the eye camera 1401 and the visual field 
cameras 1501 and 1502. Based on this signal, the subject at which the user is gazing is 
searched for with the subject candidate specific device 1702 among the drawings currently 
displayed on the display screen. On the other hand, in the input analysis device 2001, the 
input from the microphone 1901 and the keyboard 1801 is analyzed using the data 2101 for 
input analysis. This result is inputted into the input signal synthesis processing unit 2202. 
Subsequently, the subject at which the user is gazing with the subject candidate using the 
subject limited data 2203 with this device is specified, it synthesizes with the signal from the 
input analysis device 2002, and a processing instruction is outputted to the computer 2301. 
Although processing of these each part is the same as that of the case of drawing 1 on the 
whole, since the number and the mark of the display screen differ from each other, there is 
a difference at the point. Hereafter, those details are explained. 
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r0031] Drawing 12 is a flow chart which shows operation of the camera-signal-processing 
device 1602. The signal from the visual field cameras 1501 and 1502 and the eye camera 
1401 is incorporated first (Step 9211), and then the position coordinate of a viewpoint is 
searched for in the screen of a monitor of a visual field camera (Step 9212). Next, the 
marks 3001-3003 which take and surround a viewpoint in the monitor display of a visual 
field camera are specified, and the color is distinguished (Step 9213). For example, it is 
specified when have been recognized as the color of a mark being red, and a viewpoint is 
in the display screen 1001. Subsequently, about the distinguished mark, the coordinates in 
the monitor display of the visual field camera of the point of four corners of a mark are 
searched for (Step 9214). From this position coordinate and the position coordinate of a 
viewpoint, the coordinates of the viewpoint on a display screen are searched for using the 
information about actual mark shape (Step 9215). This processing is carried out with the 
technique of the usual image processing. 

[0032]Subsequently, it asks for the candidate of the subject at which the user is gazing from 
the data about the drawing currently displayed on the display screen specified with the 
subject candidate specific device 1702 when there was a viewpoint. Processing here is the 
same as processing with the object specifying device 1701 of drawing 1 mentioned above. 
However, the contents of tabulated drawing data differed and it has allowed matching two 
or more subjects with the point in the same screen. That is, a subject candidate specific 
device will output one or more subject candidates from the coordinates of a viewpoint. 
Below, the drawing of the pressure vessel of a nuclear power plant shall be displayed on 
the display screen, and two, a steam separator and a steam dome, should be obtained as a 
subject candidate. 

[0033]Next, the input from the microphone 1901 or the keyboard 1801 is analyzed with the 
input analysis device 2001 . This performs the completely same processing as the thing of 
drawing 1 . Then, considering the case where there is the input "pressure trend ****" from 
the microphone 1901 now, this input is analyzed and it is identified being a command called 
a trend display (a pressure, ?Place). Here, ?Place is a variable showing a place. 
Incorporating the candidate of this identified command and the subject searched for 
previously, the input signal synthesis processing unit 2202 creates a command to the 
computer 2301. 

[0034] Drawing 13 is a flow chart which shows operation of the input signal synthesis 
processing unit 2202. This device reads a function name and an argument value from the 
input analysis device 2001 first (Step 9221). Next, it is judged whether variable ?Place is in 
an argument (Step 9222). If there is no variable ?Place as a result of a judgment, it will 
output to the computer 2301 as it is by making the function from the input analysis device 
2001, and an argument into a processing instruction (Step 9223). On the other hand, when 
there is a variable, the subject candidate at whom the user is gazing is read from the 
subject candidate specific device 1702 (Step 9224). In now, two, a steam separator and a 
steam dome, are obtained as a subject candidate. Subsequently, subject limited data is 
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searched based on the value of a specific argument (it is the quantity of state in the case of 
now, and a value is a "pressure") (Step 9225). 

[0035]Here, the subject limited data 2203 is explained. Drawing 14 is a figure showing 
some contents of subject limited data, and the relation between quantity of state and the 
part in a plant is given. As for this data, the pressure is measured, for example about 
drywell and a steam dome. 

About the reactor core, it means that the flow is measured. 

[0036]After returning to drawing 13 and searching this data with Step 9225, it is judged 
whether there is any subject candidate as a part corresponding to an argument value (the 
present example "pressure") (Step 6226). As a result, if there is nothing, the input about a 
place will be required of a user (Step 6229). According to this demand, if there is an input in 
a microphone etc. from a user, it will be analyzed and a processing instruction will be 
completed. On the other hand, when there is a candidate, it is judged whether it is one 
(Step 6227). As a result, when there are two or more candidates, selection is required of a 
user (Step 6230). When the number of candidates is one, it is set up as a value of 
variable ?Place, a command is completed, and it outputs to a computer (Step 6228). In the 
example considered now, although two, a steam separator and a steam dome, were 
obtained as a candidate of a subject, the steam dome which is the part where the pressure 
is measured is chosen from the argument value "pressure" produced by analyzing the input 
from a microphone by reference of subject limited data. As a result, a command trend 
display (a pressure, a steam dome) 
It completes and outputs to the computer 2301. 

[0037]lf the device which becomes this example is used as explained above, when using 
two or more display screens, it can be specified in which portion of which display screen a 
user's viewpoint is. Therefore, it is effective in the command input in cases, such as a plant 
operation which uses many display screens. In this example, although the user shows one 
person's example, processing that it is the same also in the case of two or more persons is 
possible. 

[0038]Although the display screens 1001-1003 are surrounded by the frame of a different 
color and the discernment is performed in the example of drawing 11 , the example which 
was made to perform this with an option is shown in drawing 15 . In the figure, the three 
display screens 1001-1003 are formed in the indicator 2901 , and the mark 3004 is formed 
around each. This mark is filled in in paints, such as a light source of infrared rays or visible 
light, or paint. Thus, a display screen is distinguishable by the number of a mark. It can 
recognize which portion of which display screen is reflected to the picture of a visual field 
camera by changing display screen right and left and the number of an up-and-down mark. 
A mark can be distinguished in not only the number and arrangement but colors, light 
source intensity and blinking periods, or those combination. By using such a mark, it 
becomes possible also about the case where a user's head rotates greatly to distinguish 
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the four directions of a screen simply. 

r0039] Drawing 16 is a block diagram showing the composition of the device which becomes 
the 3rd example of this invention. The arrow mark 3302 is attached to the user's 2401 hat 
3301 , and the monitor camera 3101 monitors this. The display screen specific device 3201 
specifies the display screen which exists in the direction which the user has turned to based 
on the signal of the arrow mark 3302 which the monitor camera 3101 caught. The camera- 
signal-processing device 1603 searches for the viewpoint within the display screen 
specified by the upper processing based on the signal of the eye camera 1401 which the 
user carried, and the visual field camera 1501, and inputs it into the computer 2301. 
[0040]Here, a mark shall not be attached to the periphery of the display screens 1001- 
1003, or its inside, but a display screen shall use the shape and the color of the frame as a 
mark. According to this method, it is not necessary to attach a mark to a display screen or 
its inside, what exists beforehand can be used as a mark, and the time and effort of 
creation, such as a display requirement, can be saved labor. The arrow of a hat can be 
monitored, a user's motion can be caught and the candidate of a display screen whom the 
user is looking at can be scolded. When the 2nd example etc. and concomitant use are 
possible for this method and there are many especially display screens, it is useful for 
improvement in the speed of that specific processing, and simplification. 
[0041]Although the shape and the color of the frame of the display screen were used as a 
mark here, if an image processing technique progresses, it is possible to specify directly the 
subject currently displayed on the portion which recognizes the drawing displayed on the 
display screen and has a user's viewpoint. 

[0042] Drawing 17 shows the 4th example of this invention. In the figure, the meter 3401- 
3404 and the switches 3601-3604 are installed in the operator control panel 3501 of a 
plant. The light from the infrared light sources 2506-2512 is led to the point and crossings 
3701-3707 by the optical fibers 3801-3807. The disclosure fibers 3901-3907 are connected 
to this point and crossing, and this fiber is arranged so that a switch and meter may be 
surrounded. The sources 2506-2512 of infrared ray emission have generated the light of 
the wavelength which is respectively different. 

[0043]On the other hand, the eye camera 1401 and the visual field camera 1501 which the 
user 2401 carried are incorporated into the computer 4201 via the data input part 4202 via 
the signal transmission apparatus 4001 and the signal receiving set 4101. This signal is 
memorized by the memory storage 4401 via the apparatus specific part 4601, the data 
recording part 4602, and the outputting part 4203. In the apparatus specific part 4601, the 
data 4402 for apparatus specific is used via the input part 4207 and the outputting part 
4208. On the other hand, if a command is inputted via the input part 4205 from the 
command input device 4301, The data retrieval part 4603 operates and the data recorded 
via the input part 4202 is read from the memory storage 4401 , and with the gestalt which 
met the input command, the outputting part 4206 is passed, and it is displayed, printed or 
copied to the output unit 4501 . 
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[0044]This example is for recording and processing automatically a motion of the viewpoint 
of the operating staff under plant operation, for example, and if started, it will process 
drawing 18 periodically. First, it is specified within which limit of the frames of the disclosure 
fiber reflected to the visual field camera (infrared camera) the viewpoint from an eye 
camera is contained. The detailed flow chart of this processing step 9301 is shown in 
drawing 19 , and this is performed by the apparatus specific part 4601 . That is, first, the 
signal from a visual field camera and an eye camera is incorporated (Step 931 1), and the 
position coordinate of the viewpoint in the monitor display of a visual field camera is 
searched for from an eye camera signal (Step 9312). Next, the frame which takes and 
surrounds a viewpoint in the monitor display of a visual field camera is specified (Step 
9313). Next, it asks for the blinking period of the infrared light source of a frame from the 
brightness information about the specified frame (Step 9314). Based on this blinking period, 
the data for apparatus specific is used and it asks for the apparatus number within the limit 
(Step 9315). As a result, only in the case of one, apparatus specifies (Step 9316) and its 
apparatus within the specified limit, and processing is ended (Step 9317). On the other 
hand, when there is two or more apparatus within the limit, the coordinates of the viewpoint 
of a within the limit are searched for by the same processing as the camera-signal- 
processing device 1602 of the example of drawing 1 1 (Step 9318). Next, apparatus is 
specified from the coordinates of a viewpoint using the data for apparatus specific (Step 
9319). 

[0045]Some data for apparatus specific used here is shown in drawing 20 . This data has 
conferred the number of within the limit apparatus, and an equipment name for every 
blinking period of the infrared light source of a frame. When there is furthermore two or 
more apparatus within the limit, each field for defining the field corresponding to each 
apparatus is approximated with a quadrangle, and the coordinates of four square vertices 
are given. Here, when defining each field, and dividing within the limit into right-and-left 2 
field, it is possible similarly to give only the x-coordinate of a division point or to define a 
field by polygon, a circle, etc. 

[0046]Specification of the display screen in which the viewpoint is contained by Step 9301 
of drawing 18 as mentioned above will remember next the information concerning names, 
such as meter which the operating staff is looking at, and a switch, etc. to be the time at the 
time to the memory storage 4401 (Step 9302). 

[0047]On the other hand, when a display command is inputted from the command input 
device 4301 , processing of drawing 21 is started (Step 9303) and the information about 
time operation of an operating staffs viewpoint memorized by the memory storage 4401 is 
outputted to the output unit 4501 in a display or other forms (Step 9304). If the output to this 
output unit has the demand from a command input device, it is parallel to processing of 
data, namely, it can be outputted as a monitor. In the device which becomes this example, 
the apparatus etc. at which the operating staff is gazing were specified by arranging a 
disclosure fiber and changing each blinking period. This specification is feasible also by 
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various kinds of methods shown in the example described above. Or it is realizable similarly 
according to recognition of the character on the face plate installed along with the shape of 
a switch and meter, or it, etc. 

[0048]According to this example, a motion of the viewpoint of the operating staff under plant 
operation is monitored, and it can record with the subject name or number at which it is 
gazing. Therefore, for example, analysis of the working range of the frequency where meter 
is seen, and a viewpoint within a time [ a certain ], etc. can be easily conducted using a 
computer, and can be efficiently employed in evaluation of an operator control panel, a 
design, etc. Although only operation of the viewpoint was monitored with this device, if a 
microphone and voice recognition equipment are used, it is possible to record similarly and 
to use also about the history of utterance of an operating staff. 

[0049] Drawing 22 is a block diagram showing the 5th example of this invention, and the 
disclosure fiber 3908 which connected the infrared light source 2513 is arranged around the 
display screen 1001. The user 2401 had IR projector (pointing laser) 4701, or has equipped 
the finger etc. One of floodlighting and OFF switch 4702 are formed in this floodlight Fixed 
installation of the infrared camera 3102 for a monitor is carried out ahead [ of the display 
screen 1001 ], and the signal is inputted into the computer 2301 via the pointing position 
specifying apparatus 5001. The pointing position furthermore inputted into the computer is 
displayed on the display screen 1001 as the pointing marker 4901 via the display control 
1204. 

[0050] Drawing 23 is a flow chart which shows the outline of processing with the device 
which becomes this example. The position on the display screen of the point which the 
laser beam of pointing laser collides and reflects on a display screen with the pointing 
position specifying apparatus 5001 from the signal of a monitor camera is first searched for 
at Step 9401. The detailed contents of this step 9401 are shown in drawing 24 , read the 
signal of the monitor camera 31 02 first (Step 241 1 ), detect the turnoff point of the pointing 
laser beam from [ from this signal ] the floodlight 4701, and search for the coordinates in 
the monitor display of that point (Step 9412). Subsequently, it asks for the formula of the 
straight line showing each neighborhood of the frame made from the disclosure fiber 3908 
on monitor display coordinates (Step 9413). Subsequently, the coordinates on the display 
screen of the point which the turnoff point, i.e., a user, pointed at from the formula of each 
neighborhood of a frame and the coordinates of the turnoff point using the information 
about actual frame shape are searched for (Step 9414). This processing is carried out with 
the technique of the usual image processing. The frame was made from this example 
around the display screen with the infrared light source and the disclosure fiber. However, 
even when there will be no frame if the relation required for conversion of the coordinates in 
monitor display and the coordinates on a display screen is prepared beforehand since a 
monitor camera is fixable, same pointing point specific processing can be carried out. 
[0051]Next, it returns to drawing 23 , the coordinates searched for at Step 9401 are inputted 
into the computer 2301, and it uses as pointing information by pointing (Step 9402). The 
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coordinates searched for at Step 9401 are inputted into the display control 1204, and it 
displays on a display screen (Step 9403). 

[0052]As stated above, according to the device which becomes this example, a pointing 
signal can be used for pointing. It is possible to constitute a device to the conventional 
technology which needs to arrange many photo detectors with a monitor camera, an IR 
projector, and an infrared light source and a disclosure fiber around a screen, and it can 
respond to enlargement of a screen easily. Although this example showed the case where 
the number of display screens was one, Also with the case where there is two or more 
these, as other examples showed, it is realizable similarly by the method of using as a mark 
the shape of the use of visible light and the mark by a paint which change the blinking 
period of an infrared light source for every display screen, or a display screen. 
[0053] 

[Effect of the lnvention]Even when working according to this invention while a user moves 
as explained above, a motion of a user's eyeball or the information about pointing can be 
used for pointing, such as a display screen, and the pointing device which does not limit a 
service level can be realized. Simplification of an input and improvement in workability can 
be further aimed at according to concomitant use with voice input etc. The subject at which 
the user gazed is specified, and it becomes possible from a motion of an eyeball to record 
with the information about time, and is effective in the behavior analysis of a plant operator 
and the evaluation data of an operator control panel being efficiently acquirable. 
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[Filing date]November 18, Heisei 8 

[Amendment 1] 

[Document to be Amended]Specification 
[ltem(s) to be Amended]The name of an invention 
[Method of Amendment]Change 
[Proposed Amendment] 

[Title of the lnvention] Plant operation monitoring device 
[Amendment 2] 

[Document to be Amended]Speciflcation 
[ltem(s) to be Amended]Claim 
[Method of AmendmenfJChange 
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[Claim 11A plant operation monitoring device which supervises a state of plant component 
equipment comprising: 

A pointing device which directs plant component equipment currently displayed on a 
monitor screen . 

A means to specify apparatus directed with this pointing device. 
A means to output a state of apparatus specified by this means . 

[Claim 2]A plant operation monitoring device which supervises a state of plant component 
equipment, comprising: 

A pointing device which directs plant component equipment currently displayed on a 
monitor screen . 

A means to specify apparatus directed with this pointing device. 

A means to output a state of apparatus specified by this means in form of following an input 
command . 

[The amendment 3] 

[Document to be Amended]Specification 
[ltem(s) to be Amended]0001 
[Method of Amendment]Change 
[Proposed Amendment] 
[0001] 

[Industrial Application] This invention relates to a plant operation monitoring device, and 
relates to a suitable plant operation monitoring device to perform operation surveillance 
especially using a pointing device . 
[Amendment 4] 

[Document to be Amended]Specification 
[ltem(s) to be Amended]0003 
[Method of Amendment]Change 
[Proposed Amendment] 

[0003]Although the thing shown in JP,60-85899,A and the Institute of Electronics, 
Information and Communication Engineers paper magazine of the above-mentioned 
conventional technology and the thing shown in JP,62-196723,A are effective in asking for 
a user's view position under the situation where an eyeball and a head move, It cannot be 
used when the position of a user's body moves greatly. What was shown in JP,62-53630,A 
and 62-53631 , An early view image and the last view image are memorized, since it is what 
amends a user's head position from it and the newest view image, a user's motion is large 
and there is a problem that processing becomes complicated, about a case so that feature 
objects may separate temporarily from the inside of a view image. When there are two or 
more display screens, the consideration about what it is specified for in which portion of 
which display screen a viewpoint is is not made. What was shown in Japanese Patent 
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Application No. No. 99532 [ 60 to ] needs to arrange many photo detectors around a 
screen, and there was a problem that it could not manufacture simple, especially about a 
large-sized display screen. The conventional pointing device did not have consideration 
about how the information pointed at is used for the operation surveillance of a plant again. 
[Amendment 5] 

[Document to be Amended]Specification 
[ltem(s) to be Amended]0004 
[Method of Amendment]Change 
[Proposed Amendment] 

[0004]The purpose of this invention is to be able to use effectively for the operation 
surveillance of a plant the information to which it pointed with the pointing device, and to 
provide a plant operation monitoring device with high surveillance efficiency . 
[Amendment 6] 

[Document to be Amended]Specification 
[ltem(s) to be Amended]0005 
[Method of Amendment]Change 
[Proposed Amendment] 
[0005] 

[Means for Solving the Probleml ln a plant operation monitoring device with which the 
above-mentioned purpose supervises a state of plant component equipment, It is attained 
by having a pointing device which directs plant component equipment currently displayed 
on a monitor screen, a means to specify apparatus directed with this pointing device, and a 
means to output a state of apparatus specified by this means. In a plant operation 
monitoring device with which the above-mentioned purpose supervises a state of plant 
component equipment again, It is attained by having a pointing device which directs plant 
component equipment currently displayed on a monitor screen, a means to specify 
apparatus directed with this pointing device, and a means to output a state of apparatus 
specified by this means in form of following an input command. 
[Amendment 7] 

[Document to be Amended]Specification 
[ltem(s) to be Amended]0006 
[Method of Amendment]Change 
[Proposed Amendment] 
[0006] 

[Functionl lf the apparatus made into the operation surveillance object of a plant is directed 
with a pointing device on a monitor screen, it will specify what apparatus this apparatus is, 
and, as for a computer, the state of this specified apparatus will be outputted. For example, 
when a "steam dome" is directed with a pointing device on a monitor screen , a computer 
displays the trend of the pressure of this steam dome on a screen. 
[Amendment 8] 
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[Document to be Amended]Specification 
[ltem(s) to be Amended]0053 
[Method of Amendment]Change 
[Proposed Amendment] 
[0053] 

[Effect of the lnvention]lt is necessary to direct the surveillance object apparatus on a 
monitor screen with a pointing device even from where, and, according to this invention, the 
operation surveillance efficiency of a plant improves to acquire the information on 
apparatus to supervise. 



[Translation done.] 
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